Robot system

ABSTRACT

Provided is a robot system including a robot, a distance image sensor that temporally continuously acquires, from above an operating space of the robot, distance image information around the operating space, and an image processing device that processes the acquired distance image information, the image processing device defining, around the operating space, a monitoring area that includes a boundary for enabling entrance into the operating space from the outside, including a storing unit that stores reference distance image information, and detecting, based on the distance image information acquired by the distance image sensor and the reference distance image information stored in the storing unit, whether a stationary object present in the monitoring area is blocking the boundary in a visual field of the distance image sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Patent Application No.2018-204895, the content of which is incorporated herein by reference.

FIELD

The present invention relates to a robot system.

BACKGROUND

There has been known a robot system that, in distance imagescontinuously acquired by a distance image sensor disposed above a worktable, sets, around an operating space of a robot, a monitoring areahaving a boundary and causes, according to detection of an operator orthe like entering the monitoring area through the boundary, the robot toperform danger avoiding operation (see, for example, PTL 1).

Japanese Unexamined Patent Application, Publication No. 2017-221985

SUMMARY

According to an aspect of the present invention, there is provided arobot system including: a robot; a distance image sensor that temporallycontinuously acquires, from above an operating space of the robot,distance image information around the operating space; and an imageprocessing device that processes the distance image information acquiredby the distance image sensor, the image processing device defining,around the operating space, a monitoring area that includes a boundaryfor enabling entrance into the operating space from an outside,including a storing unit that stores reference distance imageinformation, which is the distance image information of the boundary ofthe monitoring area in a state in which an object is absent in themonitoring area, and detecting, based on the distance image informationacquired by the distance image sensor and the reference distance imageinformation stored in the storing unit, whether a stationary objectpresent in the monitoring area is blocking the boundary in a visualfield of the distance image sensor.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an overall configuration diagram showing a robot systemaccording to an embodiment of the present invention.

FIG. 2 is a side view showing a robot system shown in FIG. 1.

FIG. 3 is a diagram showing an example of a distance image of an outersurface of a monitoring area and an example of a distance image of abottom surface of the monitoring area acquired by the robot system shownin FIG. 1.

FIG. 4 is a side view showing an example of a case in which a stationaryobject is disposed in the monitoring area in the robot system shown inFIG. 2.

FIG. 5 is a diagram showing an example of a distance image of the outersurface of the monitoring area and an example of a distance image of thebottom surface of the monitoring area in the case of FIG. 4.

FIG. 6 is a diagram showing a differential image between the distanceimage shown in FIG. 5 and the distance image shown in FIG. 3.

FIG. 7 is a perspective view showing an example of a blind spot formedby the stationary object disposed in the monitoring area shown in FIG.4.

DETAILED DESCRIPTION

A robot system 1 according to an embodiment of the present invention isexplained below with reference to the drawings.

The robot system 1 according to this embodiment includes, as shown inFIG. 1, a robot 2, a security camera (a distance image sensor) 3disposed above an operating range (an operating space) G of the robot 2,and an image processing device 4 connected to the security camera 3.

In an example shown in FIG. 1, the robot 2 is, for example, a six-axisarticulated type robot and includes a spherical operating range G.

As the security camera 3, a stereo camera or a distance image sensor ofa TOF (Time of flight) type can be used. The security camera 3 canacquire, for each of pixels, a distance image (distance imageinformation) obtained by detecting a distance from a lens center of thesecurity camera 3.

As shown in FIG. 1 and FIG. 2, the security camera 3 has a field of viewV in which the security camera 3 is capable of photographing a rangesurrounding the entire circumference of the operating range G of therobot 2 without interfering with the operating range G of the robot 2.That is, the field of view V of the security camera 3 radially extendsfrom the lens center and expands to an annular range not including theoperating range G of the robot 2. The distance image acquired by thesecurity camera 3 is, for example, as shown in FIG. 3. In the figure, aregion indicated by hatching is the distance image.

The image processing device 4 is configured by a processor and a memory.The image processing device 4 defines a monitoring area A as shown inFIG. 2 in the distance image acquired by the security camera 3. Themonitoring area A is, for example, a square annular region having arectangular fixed cross-sectional shape and, as shown in FIG. 2,disposed in a position surrounding the entire circumference on the outerside of the operating range G of the robot 2. In FIG. 3, the distanceimage corresponds to an outer surface A1 of the monitoring area A and abottom surface A2 of the monitoring area A shown in FIG. 2.

The security camera 3 acquires, concerning the pixels, distanceinformation to an object present in the field of view V. However, aregion further on the outer side than the outer surface (a boundary) A1of the monitoring area A and a region further on the lower side than thebottom surface A2 of the monitoring area A are excluded from amonitoring target depending on the size of the distance information.

That is, a point B1 present further on the outer side than the outersurface A1 of the monitoring area A shown in FIG. 2 is disposed in aposition farther than the outer surface A1 of the monitoring area A on astraight line L1 connecting the point B1 and the lens center of thesecurity camera 3. Therefore, distance information of a pixelcorresponding to the point B1 in the distance image is larger thandistance information of a point B2 on the outer surface A1 of themonitoring area A corresponding to the same pixel. Therefore, the imageprocessing device 4 can exclude the point B1 from the monitoring targetconsidering that the point B1 is a point outside the monitoring area A.

A point C1 present further on the lower side than the bottom surface A2of the monitoring area A shown in FIG. 2 is disposed in a positionfarther than the bottom surface A2 of the monitoring area A on astraight line L2 connecting the point C1 and the lens center of thesecurity camera 3. Therefore, distance information of a pixelcorresponding to the point C1 in the distance image is larger thandistance information of a point C2 on the bottom surface A2 of themonitoring area A corresponding to the same pixel. Therefore, the imageprocessing device 4 can exclude the point C1 from the monitoring targetconsidering that the point C1 is a point outside the monitoring area A.

A stationary object (see FIG. 5) X present in the monitoring area A, forexample, an object maintained in a standstill state such as a table or acontrol device is excluded from the monitoring target by being set asthe stationary object X.

In this case, in this embodiment, the image processing device 4 detects,using a distance image obtained by the security camera 3, whether thestationary object X present in the monitoring area A is blocking theouter surface (the boundary) A1 of the monitoring area A in the field ofview (the visual field) V of the security camera 3.

Specifically, the image processing device 4 includes a storing unit (notshown in the figure) that stores a first reference distance image, whichis a distance image of each of the outer surface A1 and the bottomsurface A2 of the monitoring area A in a state in which the stationaryobject X is not disposed. The image processing device 4 can detect thestationary object X from a difference between the distance imageacquired by the security camera 3 and the reference distance image.

When the stationary object X is detected, the image processing device 4detects whether the detected stationary object X hangs over a part of aregion that should originally be the outer surface A1 or an innersurface A3 of the monitoring area A.

An example of the reference distance image is shown in FIG. 3. A sideview of the stationary object X disposed in the monitoring area A isshown in FIG. 4. An example of a distance image of the outer surface A1of the monitoring area A and an example of a distance image of thebottom surface A2 of the monitoring area A are shown in FIG. 5. Adifferential image between the reference distance image shown in FIG. 3and the distance image shown in FIG. 5 is shown in FIG. 6.

A hatched region in the monitoring area A in FIG. 4 is a blind spotformed in the monitoring area A by the stationary object X. In anexample shown in FIG. 4, the blind spot is formed to partially hang overa part of the outer surface A1 of the monitoring area A.

In FIG. 6, a hatched region indicates a region where the outer surfaceA1 of the monitoring area A is blocked by the stationary object X.

In this region, since the stationary object X is disposed in a positioncloser to the security camera 3 than the outer surface A1 of themonitoring area A, distance information smaller than the distanceinformation of the outer surface A1 is acquired in pixels in theseregions. Consequently, it is possible to detect that the outer surfaceA1 of the monitoring area A is blocked by the stationary object X.

That is, when the field of view V is blocked because the stationaryobject X hangs over the region that should be the outer surface A1 ofthe monitoring area A, there is possibility that the operator or thelike enters, without being detected by the distance image sensor, themonitoring area A from a blocked outer surface A1 portion through ablind spot hidden by the stationary object X.

It is possible to grasp presence or absence of the possibility of theentrance through the blind spot formed by the stationary object X bydetecting presence or absence of the outer surface A1 blocked by thestationary object X. When it can be determined that there is thepossibility of the entrance, it is possible to apply measures forpreventing the entrance of the operator or the like by moving thestationary object X to a position where the stationary object X does notblock the outer surface A1 or the inner surface A3 in the field of viewV or providing, on the outer side of the outer surface A1 of themonitoring area A, a wall for closing, in advance, a path leading to theblind spot.

In this way, with the robot system 1 according to this embodiment, it isdetected whether the field of view V of the security camera 3 is blockedin the region overlapping the outer surface A1 of the monitoring area Aby the stationary object X disposed in the monitoring area A. Therefore,there is an advantage that it is possible to grasp presence or absenceof possibility of entrance of the operator or the like through the blindspot formed by the stationary object X.

Note that, in this embodiment, the image processing device 4 maycalculate size of a space blocked by the stationary object X. The robotsystem 1 may include an informing unit that, when the calculated size ofthe space exceeds a predetermined threshold, informs to that effect. Asthe informing unit, it is possible to adopt a unit that informs with anymethod such as sound, light, or screen display.

Examples of the size of the space include, as shown in FIG. 7, acapacity S of a space (a hatched region) blocked by the stationaryobject X in the monitoring area A, a minimum cross-sectional area D ofthe space, and a minimum dimension E of a cross section of the space. InFIG. 7, the space forming the blind spot is a space added with a spacebelow the bottom surface A2 of the monitoring area A.

When the stationary object X is blocking the field of view V of thesecurity camera 3 on the inner surface A3 of the monitoring area A andthe space has a capacity larger than a capacity for enabling a human tohide, a human hiding in the blind spot by the stationary object X beforethe start of the robot system 1 can enter the operating range G of therobot 2 without being detected by the security camera 3. Therefore, thecapacity S of the space needs to be kept sufficiently small.

When the minimum cross-sectional area D of the space or the minimumdimension E of the cross section of the space has size for enabling ahuman or a part of the body of the human, for example, an arm or a legto pass, the human or the part of the human can enter the operatingrange G of the robot 2 without being detected by the security camera 3.Therefore, the minimum cross-sectional area D of the space or theminimum dimension E of the cross section of the space also needs to bekept sufficiently small.

The image processing device 4 may include a display unit. The displayunit is a monitor. The display unit displays the space blocked by thestationary object X. The space may be displayed in a different color inthe distance image.

The image processing device 4 may include a three-dimensional-modelgenerating unit that generates a three-dimensional model of the space.The display unit may display the generated three-dimensional model ofthe space.

As the three-dimensional model of the space, a three-dimensional modelof the space itself may be directly generated. Alternatively, bygenerating a three-dimensional model of the monitoring area A, the spaceblocked by the stationary object X may be indirectly displayed accordingto presence or absence of a region where the three-dimensional model ofthe monitoring area A is lost.

By displaying the three-dimensional model, it is possible to confirm theshape of the space blocked by the stationary object X while varying aviewing angle.

The robot system 1 may include a photographing unit that photographs therobot 2 and the periphery of the robot 2. The display unit maysuperimpose and display the three-dimensional model on an image acquiredby the photographing unit. Consequently, since the blind spot formed bythe stationary object X is displayed on the display unit as athree-dimensional model to be superimposed and displayed on an image ofthe robot 2 or the like actually acquired by the photographing unit, itis easy to grasp the position of the blind spot with respect to therobot 2 or the like. It is possible to easily apply, to the blind spot,measures for preventing entrance.

When the robot system 1 includes two or more security camera 3 and aspace blocked by the stationary object X with respect to any onesecurity camera 3 is defined as a blind spot, a sum of sets of spaces ofblind spots calculated concerning the security cameras 3 only has to becalculated as the blind spot. When a space blocked by the stationaryobject X with respect to all the security cameras 3 is defined as ablind spot, a product set of spaces of blind spots calculated concerningthe security cameras 3 only has to be calculated as the blind spot.

From the above-described embodiment, the following invention is derived.

According to an aspect of the present invention, there is provided arobot system including: a robot; a distance image sensor that temporallycontinuously acquires, from above an operating space of the robot,distance image information around the operating space; and an imageprocessing device that processes the distance image information acquiredby the distance image sensor, the image processing device defining,around the operating space, a monitoring area that includes a boundaryfor enabling entrance into the operating space from an outside,including a storing unit that stores reference distance imageinformation, which is the distance image information of the boundary ofthe monitoring area in a state in which an object is absent in themonitoring area, and detecting, based on the distance image informationacquired by the distance image sensor and the reference distance imageinformation stored in the storing unit, whether a stationary objectpresent in the monitoring area is blocking the boundary in a visualfield of the distance image sensor.

According to this aspect, when the distance image information around theoperating space is temporally continuously acquired from above theoperating space of the robot by the distance image sensor, the acquireddistance image information is processed by the image processing device.The image processing device defines, around the operating space of therobot, the monitoring area including the boundary for enabling entranceinto the operating space from the outside. Consequently, when anoperator or the like enters the monitoring area across the boundary,distance information in any pixel in the distance image informationchanges over time. Consequently, it is possible to detect the entranceof the operator or the like into the monitoring area.

On the other hand, when the stationary object is disposed in themonitoring area, a change over time of the distance information in thedistance image information does not occur. The stationary object is notdetected as an entering object.

In this case, according to this aspect, the image processing devicestores, in the storing unit, the reference distance image information,which is the distance image information of the boundary of themonitoring area in the state in which an object is absent in themonitoring area, and detects, based on the reference distance imageinformation and the distance image information acquired by the distanceimage sensor, whether the stationary object present in the monitoringarea is blocking the boundary in the visual field of the distance imagesensor.

When the stationary object is blocking the boundary, there ispossibility that the operator or the like can enter, without beingdetected by the distance image sensor, the monitoring area from ablocked boundary portion through a blind spot hidden by the stationaryobject. That is, if the operator or the like enters the monitoring areafrom the boundary portion blocked by the stationary object through theblind spot, a change over time of the distance information in thedistance image information does not occur. The entrance of the operatoror the like cannot be detected. Therefore, it is possible to grasppresence or absence of the possibility of the entrance through the blindspot formed by the stationary object by detecting presence or absence ofthe boundary blocked by the stationary object. When it can be determinedthat there is the possibility of the entrance, it is possible to applymeasures for preventing the entrance.

In the aspect, the image processing device may include an informing unitthat, when the image processing device determines that the stationaryobject is blocking the boundary, calculates size of a space blocked bythe stationary object and informs when the calculated size of the spaceexceeds a predetermined threshold.

With this configuration, it is possible to determine whether theoperator or the like can pass or hide in the blind spot formed by thestationary object. When the size of the space exceeds the predeterminedthreshold, the informing unit informs to that effect. When the informingunit informs to that effect, it is possible to apply measures forpreventing the entrance.

In the aspect, the image processing device may calculate a minimumcross-sectional area of the space as the size of the space.

With this configuration, it is possible to determine whether theoperator or the like can pass or cause a part of the body to enter theblind spot formed by the stationary object. When the size of the spaceexceeds the predetermined threshold, the informing unit informs to thateffect. When the informing unit informs to that effect, it is possibleto apply measures for preventing the entrance.

In the aspect, the image processing device may calculate a minimumdimension of a cross section of the space as the size of the space.

With this configuration, it is possible to determine whether theoperator or the like can pass or cause a part of the body to enter theblind spot formed by the stationary object. When the size of the spaceexceeds the predetermined threshold, the informing unit informs to thateffect. When the informing unit informs to that effect, it is possibleto apply measures for preventing the entrance.

In the aspect, the image processing device may include a display unitthat, when the image processing device determines that the stationaryobject is blocking the boundary, displays a space blocked by thestationary object.

With this configuration, since the blind spot formed by the stationaryobject is displayed on the display unit, it is possible to easily applymeasures for preventing entrance to the displayed blind spot.

In the aspect, the image processing device may include athree-dimensional-model generating unit that generates athree-dimensional model of the space, and the display unit may displaythe three-dimensional model of the space generated by thethree-dimensional-model generating unit.

With this configuration, since the blind spot formed by the stationaryobject is displayed on the display unit as a three-dimensional model, itis possible to change and display an angle of the blind spot and easilyapply, to the blind spot, measures for preventing the entrance.

In the aspect, the robot system may include a photographing unit thatphotographs the robot and a periphery of the robot, and the display unitmay superimpose and display the three-dimensional model on an imageacquired by the photographing unit.

With this configuration, since the blind spot formed by the stationaryobject is displayed on the display unit as a three-dimensional model tobe superimposed and displayed on an image of the robot or the likeactually acquired by the photographing unit, it is easy to grasp theposition of the blind spot with respect to the robot or the like. It ispossible to easily apply, to the blind spot, measures for preventingentrance.

The invention claimed is:
 1. A robot system comprising: a robot; and adistance image sensor that temporally continuously acquires, from abovean operating space of the robot, distance image information around theoperating space, wherein a monitoring area is defined around theoperating space which includes a boundary for enabling entrance into theoperating space from an outside area, wherein the monitoring area has asquare-annular shape with a rectangular fixed cross-section, which isdescribed by an outer surface and a lower surface such that regionsfurther on an outer side than the outer surface and further on a lowerside than the lower surface are excluded from a monitoring target,wherein reference distance image information is stored in a storingunit, which is the distance image information of the boundary of themonitoring area in a state in which an object is absent in themonitoring area, and wherein, based on the distance image informationacquired by the distance image sensor and the reference distance imageinformation stored in the storing unit, a presence of a stationaryobject located in the monitoring area which is blocking the boundary ina visual field of the distance image sensor is detected.
 2. The robotsystem according to claim 1, wherein a size of a space blocked by thestationary object is calculated and, when the calculated size of thespace exceeds a predetermined threshold, a notice is output.
 3. Therobot system according to claim 2, wherein a minimum cross-sectionalarea of the space as the size of the space is calculated.
 4. The robotsystem according to claim 2, wherein a minimum dimension of a crosssection of the space as the size of the space is calculated.
 5. Therobot system according to claim 1, wherein a display unit displays aspace blocked by the stationary object.
 6. The robot system according toclaim 5, wherein the display unit displays a three-dimensional model ofthe space.
 7. The robot system according to claim 6, further comprisinga photographing unit that photographs the robot and a periphery of therobot, wherein the display unit superimposes and displays thethree-dimensional model on an image acquired by the photographing unit.8. The robot system according to claim 1, wherein the robot systemcomprises a plurality of distance image sensors, wherein a size of aproduct set of a space blocked by the stationary object with respect toeach of the plurality of distance image sensors is calculated, and whenthe calculated size of the product set exceeds a predeterminedthreshold, a notice is output.